Object detection and identification

ABSTRACT

An embodiment of the invention may include a method, computer program product and computer system for object detection and identification. The method, computer program product and computer system may include computing device which may receive an image from a user device. The image may be a screenshot captured by the user device from a display. The computing device may classify the image based on features present in the image and detect a salient object contained within the image. The computing device may identify the object in the image and one or more sources of the object in the image.

BACKGROUND

The present invention relates generally to a method, system, andcomputer program for object detection and identification. Moreparticularly, the present invention relates to a method, system, andcomputer program for detecting an object contained within an image andidentifying the object along with sources of the object.

Humans are capable of looking at an image or watching a video andreadily identifying, people, objects, scenes, and other visual details.Object recognition has become an ever increasingly important facet ofmodern technology. Object recognition, with respect to technology, is acomputer vision technique for identifying objects in images or videos.Object recognition techniques may use various means to identify objectssuch as deep learning and machine learning algorithms. Further, objectrecognition techniques may be combined with object detection techniques.Object detection and object recognition are similar techniques foridentifying objects, but they vary in their execution. Object detectionis the process of finding instances of objects in images. In the case ofdeep learning, object detection is a subset of object recognition, wherethe object is not only identified but also located in an image. Thisallows for multiple objects to be identified and located within the sameimage.

BRIEF SUMMARY

An embodiment of the invention may include a method, computer programproduct and computer system for object detection and identification. Themethod, computer program product and computer system may includecomputing device which may receive an image from a user device. Theimage may be a screenshot captured by the user device from a display.The computing device may classify the image based on features present inthe image and detect a salient object contained within the image. Thecomputing device may identify the object in the image and one or moresources of the object in the image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a illustrates a system for object detection and identification, inaccordance with an embodiment of the invention.

FIG. 1b illustrates example operating modules of the object detectionand identification program of FIG. 1 a.

FIG. 2a is a flowchart illustrating an example method of objectdetection and identification in accordance with an embodiment of theinvention.

FIG. 2b is a flowchart illustrating an example method of objectdetection and identification in accordance with an embodiment of theinvention.

FIG. 3 is a block diagram depicting the hardware components of theobject detection and identification system of FIG. 1, in accordance withan embodiment of the invention.

FIG. 4 illustrates a cloud computing environment, in accordance with anembodiment of the invention.

FIG. 5 illustrates a set of functional abstraction layers provided bythe cloud computing environment of FIG. 4, in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described in detailwith reference to the accompanying Figures.

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used to enablea clear and consistent understanding of the invention. Accordingly, itshould be apparent to those skilled in the art that the followingdescription of exemplary embodiments of the present invention isprovided for illustration purpose only and not for the purpose oflimiting the invention as defined by the appended claims and theirequivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces unless the context clearly dictatesotherwise.

Advertisements for products are ubiquitous in today's society withvarious products being advertised on billboards, commercials, anddigital ads on the internet and IOT applications, etc. Further, productsare also advertised using product placement wherein products areincorporated into television programs and movies, etc. Productplacements may range from unobtrusive appearances within an environmentto prominent integration and acknowledgement of the product within thework. Products included in product placements may include automobiles,consumer electronics, clothing, personal accessories, shoes, jewelry,and food, just to name a few. Thus, there may be instances wherein aconsumer views a television program or movie and sees a product they maylike and wish to acquire.

Embodiments of the present invention provide a method, computer program,and computer system for detecting an object contained within an imageand identifying the object along with sources of the object and/orsources related to the object. Embodiments of the present invention alsoprovide a method, computer program, and computer system for displayingan object on an image of the user and providing a means for purchasing,renting, borrowing, or otherwise acquiring the object. Moreparticularly, embodiments of the present invention receive an image froma display, analyze the image for retail or non-retail objects andgenerate a list of sources of the retail or non-retail objects forpresentation to a user. Advantages of the invention over currenttechnology include saliency detection of objects within images,delamination of retail objects and non-retail objects, multiple sourceidentification based on location of a user, image creation of the userwith the identified objects, and object acquisition verification using abiometric sensor.

Reference will now be made in detail to the embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to like elementsthroughout. Embodiments of the invention are generally directed to asystem for object detection and identification.

FIG. 1 illustrates an object detection and identification system 100, inaccordance with an embodiment of the invention. In an exampleembodiment, object detection and identification system 100 includes adisplay device 110, a user device 120, and server 130, interconnectedvia network 140.

In the example embodiment, the network 140 is the Internet, representinga worldwide collection of networks and gateways to supportcommunications between devices connected to the Internet. The network140 may include, for example, wired, wireless or fiber opticconnections. In other embodiments, the network 140 may be implemented asan intranet, a local area network (LAN), or a wide area network (WAN).In general, the network 140 can be any combination of connections andprotocols that will support communications between the display device110, the user device 120, and the server 130.

The display device 110 may include the image database 112. The displaydevice 110 may be any device capable of displaying the image data 114.The image data 114 may include, but is not limited to, visual, audio,and/or textual data. For example, the display device 110 may displayvideo or images, or both, such as, but not limited to, televisionprograms, commercials, movies, pictures, billboards, digital ads,notices, announcements, and advertisements, etc. Further, the image data114 may be, but is not limited to screenshots of the visual, audio,and/or textual data. The video and images displayed by the displaydevice 110 may also contain textual data, such as, but not limited to,object names, object source identifiers, and object brand names, etc. Inthe example embodiment, the display device 110 may be a television, amovie screen, a projector, a camera, a computer, a tablet, a thinclient, a cellphone, or any other device capable of displaying visual,audio, and/or textual data and sending that visual, audio, and/ortextual data to and from other computing devices, such as the userdevice 120, and the server 130 via the network 140. The display device110 is described in more detail with reference to FIG. 3.

The image database 112 may store the image data 114, i.e. the visual,audio, and/or textual data, being displayed by the display device 110.The image database 112 may be any storage media capable of storing datacapable of storing data, such as, but not limited to, storage mediaresident in the display device 110 and/or removeable storage media. Forexample, the image database 112 may be, but is not limited to, a harddrive, a solid stated drive, a USB drive, or a memory card, etc. Theimage database 112 is described in more detail above and with referenceto FIG. 3.

The user device 120 may include the user interface 122. In the exampleembodiment, the user device 120 may be a cellphone, desktop computer, anotebook, a laptop computer, a tablet computer, a thin client, or anyother electronic device or computing system capable of storing compilingand organizing audio, visual, or textual content and receiving andsending that content to and from other computing devices, such as thedisplay device 110, and the server 130 via the network 140. While only asingle user device 120 is depicted, it can be appreciated that anynumber of user devices may be part of the object detection andidentification system 100. In embodiments of the invention, the displaydevice 110 and the user device 120 may be the same device. For example,a user may watch a television program or a movie on the user device 120,e.g. their cellphone or computer, and capture images of the televisionprogram or movie from the user device 120, e.g. their cellphone orcomputer. In some embodiments, the user device 120 includes a collectionof devices or data sources. The user device 120 is described in moredetail with reference to FIG. 3.

The user interface 122 includes components used to receive input from auser on the user device 120 and transmit the input to the objectdetection and identification program 136 residing on server 130, orconversely to receive information from the object detection andidentification program 136 and display the information to the user onuser device 120. In an example embodiment, the user interface 122 uses acombination of technologies and devices, such as device drivers, toprovide a platform to enable users of the user device 120 to interactwith the object detection and identification program 136. In the exampleembodiment, the user interface 122 receives input, such as but notlimited to, textual, visual, or audio input received from a physicalinput device, such as but not limited to, a keypad and/or a microphone.For example, the user interface 122 may include, but is not limited to,a button or an image of a button, which the user may touch to capture ascreenshot of the image data 114 displayed in the display device 110.

The server 130 may include the program database 132 and the objectdetection and identification program 136. In the example embodiment, theserver 130 may be a desktop computer, a notebook, a laptop computer, atablet computer, a thin client, or any other electronic device orcomputing system capable of storing compiling and organizing audio,visual, or textual content and receiving and sending that content to andfrom other computing devices, such as the display device 110, and theuser device 120 via network 140. In some embodiments, the server 130includes a collection of devices, or data sources, in order to collectthe program data 134. The server 130 is described in more detail withreference to FIG. 3.

The program database 132 may store the program data 134. The programdatabase 132 may be any storage media capable of storing data, such as,but not limited to, storage media resident in the server 130 and/orremoveable storage media. For example, the program database 132 may be,but is not limited to, a hard drive, a solid stated drive, a USB drive,or a memory card, etc. The program database 132 is described in moredetail above and with reference to FIG. 3.

The program data 134 may be a collection of audiovisual contentincluding, but not limited to, audio, visual, and textual content. Theprogram data 134 may be, for example, the image data 114 received and/orcollected from the display device 110 and the user device 120. Further,the program data 134 may include user data such as, but not limited to,a user's identification, a user's phone number, a user's address, auser's preferences, a list of the user device 120 associated with auser, and the history of user interactions with the object detection andidentification program 136, and photographs of the user, etc. Theprogram data 134 is located on the server 130 and can be accessed viathe network 140. In accordance with an embodiment of the invention, theprogram data 134 may be located on one or a plurality of servers 130.

The object detection and identification program 136 is a program capableof receiving the image data 114 captured by the user device 120 andanalyzing the image data 114 to detect and identify objects containedwithin the image data 114. In some embodiments of the invention, theobject detection and identification program 136 may generate an image ofan identified object on the user. Further, embodiments of the objectdetection and identification program 136 may also detect a user'slocation, identify sources of any identified object, and facilitate theacquisition of any identified object. The object detection andidentification program 136 is described in more detail below withreference to FIG. 1 b.

FIG. 1b illustrates example modules of the object detection andidentification program 136. In an example embodiment, the objectdetection and identification program 136 may include eight modules:image capture module 150, image classification module 152, objectdetection module 154, object identification module 156, sourceidentification module 158, user notification module 160, objectvisualization module 162, and object acquisition module 164.

The image capture module 150 receives the image data 114 captured fromthe display device 110 by the user device 120. In an embodiment of theinvention, the image data 114 may be stored as the program data 134 onthe program database 132. For example, but not limited to, the userdevice 120 may capture a screenshot of a movie or television programbeing displayed on the display device 110 using the user interface 122.The screenshot of the movie or television program being displayed on thedisplay device 110 would then be sent to the server 130 where it wouldbe received by the image capture module 150 of the object detection andidentification program 136 via the user interface 122.

The image classification module 152 classifies the image data 114received by the image capture module 150. The image classificationmodule 152 may utilize machine algorithms to classify the image data 114received by the image capture module 150 according to various metricssuch as, but not limited to, scenes, geography, objects, people, faces,colors, food, text, etc. Further, the image classification module 152may utilize metadata associated with the image data 114 received by theimage capture module 150. The image data 144 may contain metadataincluding, but not limited to, the time the image data 114 was captured,the date the image data 114 was taken, and where the image data 114 wastaken, etc. In an example embodiment, the image classification module152 may utilize a machine algorithm to classify the image data 114 usinglandmarks and features contained within the image data 114. As anon-limiting example, the image data 114 may contain geographical imagesrepresenting a country, a party, a beach, a mountain, etc., and amachine algorithm would be able to further classify the image data 114according to the location the image data 114 was captured. The imageclassification module 152 may also utilize a machine algorithm toclassify the image data 114 based on elements contain within the imagedata 114. Thus, the image classification module 152 may classify theimage data 114 based on, for example, but not limited to, a store, classroom, a childcare facility room, or a sport field or facility featuredin the image data 114, etc. Further, the image classification module 152may utilize a machine algorithm to classify the image data 114 based onpeople or animals depicted in the image data 114 such as, but notlimited to, an actor, an actress, a spokesperson, a mascot, gender, andage, etc.

The object detection module 154 detects objects contained within theimage data 114 received by the image capture module 150. The objectidentification module 156 detects identifiable objects contained withinthe image data 114. Further, the object identification module 156 maycategorize the identifiable objects such as, but not limited toautomobiles, consumer electronics, clothing, personal accessories,shoes, jewelry, and food, etc. The object identification module 156 mayuse one or more object recognition techniques such as, but not limitedto, saliency detection and/or visual quantification to detect andcategorize objects contained within the image data 114 received by theimage capture module 150. For example, the object recognition technologymay be, but not limited to, a trained object detection model. Thetrained object detection model may be generated using neural networks,including, but not limited to, deep convolutional neural networks, anddeep recurrent neural networks. Deep convolutional neural networks are aclass of deep, feed-forward artificial neural networks consisting of aninput layer, an output layer, and multiple hidden layers used to analyzeimages. Deep recurrent neural networks are artificial neural networkswherein the connections between the nodes of the network form a directedgraph along a sequence used for analyzing linguistic data. The objectdetection module 154 may input the image data 114 into the convolutionalneural networks to generate the trained object detection model. Thetrained object detection model detects unique objects contained withinthe image data 114. As another example, the object recognitiontechnology may include, but it not limited to, a saliency detectionalgorithm such as SalNet. SalNet is a deep learning algorithm whichautomatically detects salients for a given image such as an objectcontained within the image data 114. The saliency of an image is thestate or quality by which it stands out relative to its neighbors, i.e.localizing what people see when they view the image. Saliency detectionis considered to be a key attentional mechanism that facilitateslearning and survival by enabling organisms to focus their limitedperceptual and cognitive resources on the most pertinent subset of theavailable sensory data. Saliency detection stresses on four types offeatures, namely color, luminance, texture, and depth. In embodiments ofthe present invention, saliency detection concentrates primarily onstatic saliency and objectness. Static saliency detection algorithms usedifferent image features that allow detecting salient object of anon-dynamic image and objectness estimation seeks to propose a small setof bounding boxes according to the possibility of a complete objectexisting around a region.

The object identification module 156 identifies one or more individualobjects detected by the object detection module 154. For example, theimage data 114 may be, for example, but not limited to, a screenshot ofa movie depicting an actor and the object identification module 156 mayidentify the individual pieces of clothing, jewelry, and/or accessoriesthe actor is wearing or using in the image. The object identificationmodule 156 may identify the one or more individual objects detected bythe object detection module 154 by delaminating, i.e. separating, theimage data 114 into retail, e.g. clothing, jewelry, personalelectronics, and furniture, etc., and non-retail objects, e.g. people,animals, public and commercial services or facilities, etc. The objectidentification module 156 may utilize multi-modal learning to identifythe one or more individual objects. For example, the multi-modallearning may include, but is not limited to, neural networks, backgroundsubtraction techniques, k-means algorithms, Barnes-Hut approximations,and/or t-Distributed Stochastic Neighbor Embedding (t-SNE), etc.

The source identification module 158 determines the location of the userdevice 120 and identifies one or more sources of the one or more objectsbased on the location of the user device 120. In some embodiments, asource of an object may be, but not limited to, a seller of the object,a lender of the object, a lessor of the object, a donor of the object,etc. In some embodiments, a source of an object may be, but is notlimited to, a provider of the object and/or a service or facilityrelated to the object. For example, a library or a bookstore may besources for a book object. As another example, a sport facility, field,or court, or a seller of sporting equipment, may be sources forsport-related object, like a racket or ball. As yet another example, theobjects identified as salient in a screenshot of pre-school agedchildren working on art projects in a daycare facility may include itemsof particular items of clothing or jewelry (e.g., shirts, pants,dresses, shoes, bracelets, necklaces, headwear) the children arewearing, the various art supplies (e.g., paper, crayons) the childrenare using, and the daycare room in which the children are present.According to various embodiments, sources of items of clothing may besellers of children's clothing, sources of the daycare room may becommercial, religious, governmental, or non-profit providers of care forchildren. According to various embodiments, sources of art supplyobjects may be sellers of art supply objects or organizations that offerart classes or instruction. The source identification module 158 may forexample, but not limited to, receive the location of the user device 120from the user device 120, and/or determine the location based on theimage data 114 received from the user device 120. The one or moresources may include, but are not limited to, e-commerce portals,retailers, churches, sports fields or facilities, libraries, schools,childcare providers, etc. The source identification module 158 may alsoidentify the prices of, or other conditions or requirements forobtaining the one or more identified objects (or service) for each ofthe one or more sources. For example, the source identification module158 may determine that a watch contained within the image data 114 isavailable for sale at two brick-and-mortar retailers near the userdevice 120 and through several e-commerce retailers such, but notlimited to, Amazon®, and eBay®, etc. Further, the source identificationmodule 158 may determine the price of the watch at each source. Asanother example, the source identification module 158 may determine anyconditions on obtaining a loan of a book object from a library, e.g., aresidency condition. As a further example, the source identificationmodule 158 may determine any conditions on obtaining art classes orinstruction for an art supply object, e.g., a minimum age or experiencelevel condition. As a further example, the source identification module158 may determine any conditions on obtaining daycare services from anorganization in connection with a daycare room being determined salient,e.g., membership in a religious organization condition or a maximumhousehold income condition. As a further example, the object detectionmodule 156 may identify a book in the image data 114 and the sourceidentification module 158 may determine sources of the book such as, butnot limited to, a library, a bookstore, an electronic book store, anelectronic book sharing website or application, etc. In furtherembodiments, the source identification module 158 may identify sources,such as, but not limited to, venues for viewing objects which cannot beacquired but may be viewed. For example, the image data 114 may be apicture of a defense aircraft in a national museum and the sourceidentification module 158 may identify a museum where such defenseaircrafts are on display.

The user notification module 160 generates a list of the one or moresources of the one or more objects based on the location of the userdevice 120 and presents the list to the user on the user device 120 viathe user interface 122. The list may also include the prices of the oneor more objects associated with the one or more sources. Thus, a user onthe user device 120 may, at any point of time, access the list, lookthrough the various offers, compare the best price against listedentries and place an order, as well choose to walk to a nearby retailoutlet which has advertised of discounts and offers, based on thegeographical location of the user device 120.

The object visualization module 162 receives a user image from the userdevice which may be stored in the program data 134 on the programdatabase 132. The object visualization module 162 generates an imagedepicting the one or more identified objects on the user image. Theobject visualization module 162 may use neural networks such as, but notlimited to convolutional neural networks (CNNs) to superimpose theidentified object onto an image of the user stored as the program data134 on the program database 132. For example, the identified object maybe a piece of jewelry and the object visualization module 162 maygenerate an image depicting the necklace on a user of the user device120 so that the user may visualize the object on themselves beforeacquiring.

The object acquisition module 164 receives a request from the userdevice 120 to acquire one of the one or more identified objects from anidentified source and sends the request to the identified source. In anembodiment, the object acquisition module 164 may verify the acquisitionrequest received from the user device 120. The object acquisition module164 may verify the acquisition or purchase using an authenticationmechanism, such as, but not limited to, a biometric sensor, a one-timepassword (OTP), a passcode, a password, etc. to confirm the placement ofan acquisition request as well complete a payment transaction.

Referring to FIGS. 2a -2 b, a method 200 for object detection andidentification is depicted, in accordance with an embodiment of thepresent invention.

Referring to block 210, the image capture module 150 receives the imagedata 114 captured from the display device 110 by the user device 120.Image capture is described in more detail above with reference to theimage capture module 150 of FIG. 1 b.

Referring to block 212, The image classification module 152 classifiesthe image data 114 received by the image capture module 150. Imageclassification is described in more detail above with reference to theimage classification module 152 of FIG. 1 b.

Referring to block 214, the object detection module 154 detects objectscontained within the image data 114 received by the image capture module150. Object detection is described in more detail above with referenceto the object detection module 154 of FIG. 1 b.

Referring to block 216, the object identification module 156 identifiesone or more individual objects detected by the object detection module154. Object identification is described in more detail above withreference to the object identification module 156 of FIG. 1 b.

Referring to block 218, the source identification module 158 determinesthe location of the user device 120. Determining user location isdescribed in more detail above with reference to the sourceidentification module 158 of FIG. 1 b.

Referring to block 220, the source identification module 158 identifiesone or more sources of the one or more objects based on the location ofthe user device 120. Source identification is described in more detailabove with reference to the source identification module 158 of FIG. 1b.

Referring to block 222, the user notification module 160 generates alist of the one or more sources of the one or more objects based on thelocation of the user device 120. Source list generation is described inmore detail above with reference to the user notification module 160 ofFIG. 1 b.

Referring to block 224, the user notification module 160 presents thelist to the user on the user device 120 via the user interface 122.Source list presentation is described in more detail above withreference to the user notification module 160 of FIG. 1 b. Following thesource list presentation, the object detection and identificationprogram 136 may continue to block 224 or terminate.

Referring to block 226, the object visualization module 162 receives auser image from the user device which may be stored in the program data134 on the program database 132. User image receipt is described in moredetail above with reference to the object visualization module 162 ofFIG. 1 b.

Referring to block 228, the object visualization module 162 generates animage depicting the one or more identified objects on the user image.Object visualization is described in more detail above with reference tothe object visualization module 162 of FIG. 1 b. Following the objectvisualization, the object detection and identification program 136 maycontinue to block 230 or terminate.

Referring to block 230, the object acquisition module 164 receives arequest from the user device 120 to acquire one of the one or moreidentified objects from an identified source. Object acquisition requestis described in more detail above with reference to the objectacquisition module 164 of FIG. 1 b.

Referring to block 232, the object acquisition module 164 verifies theacquisition request received from the user device 120. Acquisitionrequest verification is described in more detail above with reference tothe object acquisition module 164 of FIG. 1 b.

Referring to block 234, the object acquisition module 164 sends therequest to the identified source. Acquisition request transmittal isdescribed in more detail above with reference to the object acquisitionmodule 164 of FIG. 1 b.

Referring to FIG. 3, a system 1000 includes a computer system orcomputer 1010 shown in the form of a generic computing device. Themethod 200 for example, may be embodied in a program(s) 1060 (FIG. 3)embodied on a computer readable storage device, for example, generallyreferred to as memory 1030 and more specifically, computer readablestorage medium 1050 as shown in FIG. 3. For example, memory 1030 caninclude storage media 1034 such as RAM (Random Access Memory) or ROM(Read Only Memory), and cache memory 1038. The program 1060 isexecutable by the processing unit or processor 1020 of the computersystem 1010 (to execute program steps, code, or program code).Additional data storage may also be embodied as a database 1110 whichcan include data 1114. The computer system 1010 and the program 1060shown in FIG. 3 are generic representations of a computer and programthat may be local to a user, or provided as a remote service (forexample, as a cloud based service), and may be provided in furtherexamples, using a website accessible using the communications network1200 (e.g., interacting with a network, the Internet, or cloudservices). It is understood that the computer system 1010 alsogenerically represents herein a computer device or a computer includedin a device, such as a laptop or desktop computer, etc., or one or moreservers, alone or as part of a datacenter. The computer system caninclude a network adapter/interface 1026, and an input/output (I/O)interface(s) 1022. The I/O interface 1022 allows for input and output ofdata with an external device 1074 that may be connected to the computersystem. The network adapter/interface 1026 may provide communicationsbetween the computer system a network generically shown as thecommunications network 1200.

The computer 1010 may be described in the general context of computersystem-executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The method steps and system components and techniques may be embodied inmodules of the program 1060 for performing the tasks of each of thesteps of the method and system. The modules are generically representedin FIG. 3 as program modules 1064. The program 1060 and program modules1064 can execute specific steps, routines, sub-routines, instructions orcode, of the program.

The method of the present disclosure can be run locally on a device suchas a mobile device, or can be run a service, for instance, on the server1100 which may be remote and can be accessed using the communicationsnetwork 1200. The program or executable instructions may also be offeredas a service by a provider. The computer 1010 may be practiced in adistributed cloud computing environment where tasks are performed byremote processing devices that are linked through a communicationsnetwork 1200. In a distributed cloud computing environment, programmodules may be located in both local and remote computer system storagemedia including memory storage devices.

More specifically, as shown in FIG. 3, the system 1000 includes thecomputer system 1010 shown in the form of a general-purpose computingdevice with illustrative periphery devices. The components of thecomputer system 1010 may include, but are not limited to, one or moreprocessors or processing units 1020, a system memory 1030, and a bus1014 that couples various system components including system memory 1030to processor 1020.

The bus 1014 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

The computer 1010 can include a variety of computer readable media. Suchmedia may be any available media that is accessible by the computer 1010(e.g., computer system, or server), and can include both volatile andnon-volatile media, as well as, removable and non-removable media.Computer memory 1030 can include additional computer readable media 1034in the form of volatile memory, such as random access memory (RAM),and/or cache memory 1038. The computer 1010 may further include otherremovable/non-removable, volatile/non-volatile computer storage media,in one example, portable computer readable storage media 1072. In oneembodiment, the computer readable storage medium 1050 can be providedfor reading from and writing to a non-removable, non-volatile magneticmedia. The computer readable storage medium 1050 can be embodied, forexample, as a hard drive. Additional memory and data storage can beprovided, for example, as the storage system 1110 (e.g., a database) forstoring data 1114 and communicating with the processing unit 1020. Thedatabase can be stored on or be part of a server 1100. Although notshown, a magnetic disk drive for reading from and writing to aremovable, non-volatile magnetic disk (e.g., a “floppy disk”), and anoptical disk drive for reading from or writing to a removable,non-volatile optical disk such as a CD-ROM, DVD-ROM or other opticalmedia can be provided. In such instances, each can be connected to bus1014 by one or more data media interfaces. As will be further depictedand described below, memory 1030 may include at least one programproduct which can include one or more program modules that areconfigured to carry out the functions of embodiments of the presentinvention. As such, the computing device in FIG. 4 becomes specificallyconfigured to implement mechanisms of the illustrative embodiments andspecifically configured to perform the operations and generated theoutputs of described herein for determining a route based on a user'spreferred environmental experiences.

The methods 200 (FIG. 2), for example, may be embodied in one or morecomputer programs, generically referred to as a program(s) 1060 and canbe stored in memory 1030 in the computer readable storage medium 1050.The program 1060 can include program modules 1064.

The program modules 1064 can generally carry out functions and/ormethodologies of embodiments of the invention as described herein. Forexample, the program modules 1064 can include the modules 150-164described above with reference to FIG. 1 b. The one or more programs1060 are stored in memory 1030 and are executable by the processing unit1020. By way of example, the memory 1030 may store an operating system1052, one or more application programs 1054, other program modules, andprogram data on the computer readable storage medium 1050. It isunderstood that the program 1060, and the operating system 1052 and theapplication program(s) 1054 stored on the computer readable storagemedium 1050 are similarly executable by the processing unit 1020.

The computer 1010 may also communicate with one or more external devices1074 such as a keyboard, a pointing device, a display 1080, etc.; one ormore devices that enable a user to interact with the computer 1010;and/or any devices (e.g., network card, modem, etc.) that enables thecomputer 1010 to communicate with one or more other computing devices.Such communication can occur via the Input/Output (I/O) interfaces 1022.Still yet, the computer 1010 can communicate with one or more networks1200 such as a local area network (LAN), a general wide area network(WAN), and/or a public network (e.g., the Internet) via networkadapter/interface 1026. As depicted, network adapter 1026 communicateswith the other components of the computer 1010 via bus 1014. It shouldbe understood that although not shown, other hardware and/or softwarecomponents could be used in conjunction with the computer 1010.Examples, include, but are not limited to: microcode, device drivers1024, redundant processing units, external disk drive arrays, RAIDsystems, tape drives, and data archival storage systems, etc.

It is understood that a computer or a program running on the computer1010 may communicate with a server, embodied as the server 1100, via oneor more communications networks, embodied as the communications network1200. The communications network 1200 may include transmission media andnetwork links which include, for example, wireless, wired, or opticalfiber, and routers, firewalls, switches, and gateway computers. Thecommunications network may include connections, such as wire, wirelesscommunication links, or fiber optic cables. A communications network mayrepresent a worldwide collection of networks and gateways, such as theInternet, that use various protocols to communicate with one another,such as Lightweight Directory Access Protocol (LDAP), Transport ControlProtocol/Internet Protocol (TCP/IP), Hypertext Transport Protocol(HTTP), Wireless Application Protocol (WAP), etc. A network may alsoinclude a number of different types of networks, such as, for example,an intranet, a local area network (LAN), or a wide area network (WAN).

In one example, a computer can use a network which may access a websiteon the Web (World Wide Web) using the Internet. In one embodiment, acomputer 1010, including a mobile device, can use a communicationssystem or network 1200 which can include the Internet, or a publicswitched telephone network (PSTN) for example, a cellular network. ThePSTN may include telephone lines, fiber optic cables, microwavetransmission links, cellular networks, and communications satellites.The Internet may facilitate numerous searching and texting techniques,for example, using a cell phone or laptop computer to send queries tosearch engines via text messages (SMS), Multimedia Messaging Service(MMS) (related to SMS), email, or a web browser. The search engine canretrieve search results, that is, links to websites, documents, or otherdownloadable data that correspond to the query, and similarly, providethe search results to the user via the device as, for example, a webpage of search results.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 4, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 4 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 5, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 4) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 5 are intended to be illustrative only and embodiments of theinvention are not limited thereto. As depicted, the following layers andcorresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and object detection and identification 96.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While steps of the disclosed method and components of the disclosedsystems and environments have been sequentially or serially identifiedusing numbers and letters, such numbering or lettering is not anindication that such steps must be performed in the order recited, andis merely provided to facilitate clear referencing of the method'ssteps. Furthermore, steps of the method may be performed in parallel toperform their described functionality.

1-8. (canceled)
 9. A computer program product for object detection andidentification, the computer program product comprising: acomputer-readable storage medium having program instructions embodiedtherewith, wherein the computer readable storage medium is not atransitory signal per se, the program instructions comprising: programinstructions to receive, by a computing device, an image from a userdevice, wherein the image is screenshot captured by the user device froma display; program instructions to classify, by the computing device,the image, wherein the image is classified based on features present inthe image; program instructions to detect, by the computing device, anobject contained within the image, wherein the object is a salientobject; program instructions to identify, by the computing device, theobject in the image; and program instructions to identify, by thecomputing device, one or more sources of the object in the image.
 10. Acomputer program product as in claim 9, wherein the program instructionsto identify, by the computing device, one or more sources of the objectin the image further comprise: program instructions to determine by thecomputing device, a location of the user device; program instructions togenerate, by the computing device, a list of sources of the object basedon the location of the user device; and program instructions to present,by the computing device, the list of sources of the object to a user onthe user device, thereby allowing the user to compare respectiveconditions and locations for each of the objects.
 11. A computer programproduct as in claim 9, further comprising: program instructions toreceive, by the computing device, a second image, the second image beingan image of the user, from the user device; and program instructions togenerate, by the computing device, a second image of the user with theobject, wherein the second image is generated using at least oneconvolutional neural network.
 12. A computer program product as in claim9, further comprising: program instructions to receive, by the computingdevice, a request to acquire the object from the user device from one ofthe one or more sources; program instructions to verify, by the computerdevice, the request to acquire the object; and program instructions tosend, by the computer device, the request to acquire the object to thesource.
 13. A computer program product as in claim 9, wherein thescreenshot is captured from the display displaying at least one of thegroup consisting of: a movie, a television program, and a commercial.14. A computer program product as in claim 9, wherein the object isidentified using multi-modal learning techniques, the multi-modallearning techniques comprising at least one of the group consisting of:a neural network, a convolutional neural network (CNN), a backgroundsubtraction technique, a k-means algorithm, a Barnes-Hut approximation,and a t-Distributed Stochastic Neighbor Embedding (t-SNE).
 15. Acomputer program product as in claim 12, wherein the request to acquirethe object is verified using a biometric sensor on the user device. 16.A computer system for object detection and identification, the systemcomprising: one or more computer processors, one or morecomputer-readable storage media, and program instructions stored on oneor more of the computer-readable storage media for execution by at leastone of the one or more processors, the program instructions comprising:program instructions to receive, by a computing device, an image from auser device, wherein the image is screenshot captured by the user devicefrom a display; program instructions to classify, by the computingdevice, the image, wherein the image is classified based on featurespresent in the image; program instructions to detect, by the computingdevice, an object contained within the image, wherein the object is asalient object; program instructions to identify, by the computingdevice, the object in the image; and program instructions to identify,by the computing device, one or more sources of the object in the image.17. A computer system as in claim 16, wherein the program instructionsto identify, by the computing device, one or more sources of the objectin the image further comprise: program instructions to determine by thecomputing device, a location of the user device; program instructions togenerate, by the computing device, a list of sources of the object basedon the location of the user device; and program instructions to present,by the computing device, the list of sources of the object to a user onthe user device, thereby allowing the user to compare respectiveconditions and locations for each of the objects.
 18. A computer systemas in claim 16, further comprising: program instructions to receive, bythe computing device, a second image, the second image being an image ofthe user, from the user device; and program instructions to generate, bythe computing device, a second image of the user with the object,wherein the second image is generated using at least one convolutionalneural network.
 19. A computer system as in claim 16, furthercomprising: program instructions to receive, by the computing device, arequest to acquire the object from the user device from one of the oneor more sources; program instructions to verify, by the computer device,the request to acquire the object, wherein the request to acquire theobject is verified using a biometric sensor on the user device; andprogram instructions to send, by the computer device, the request toacquire the object to the source.
 20. A computer system as in claim 16,wherein the object is identified using multi-modal learning techniques,the multi-modal learning techniques comprising at least one of the groupconsisting of: a neural network, a convolutional neural network (CNN), abackground subtraction technique, a k-means algorithm, a Barnes-Hutapproximation, and a t-Distributed Stochastic Neighbor Embedding(t-SNE).